A scoping review of Do-It-Yourself Automated Insulin Delivery system (DIY AID) use in people with type 1 diabetes

Introduction User designed Automated Insulin Delivery systems (AID), termed Do-It-Yourself (DIY) AID include; AndroidAPS, OpenAPS and Loop. These unregulated systems provide challenges for healthcare providers worldwide, with potential legal and ethical barriers to supporting their use. We performed a scoping review of the currently available literature surrounding DIY AID systems, specifically to highlight the evidence available to facilitate healthcare providers to support persons with diabetes who may benefit from DIY AID. Methods Studies relating to DIY AID systems were searched in Embase, Medline, Web of Science, Scopus, Proquest and Cochrane library until 31st December 2021. Publications were screened through title and abstract to identify study type and AID system type described. A thematic synthesis methodology was used for analysis of studies of DIY AID use due to the heterogeneity in study designs (case reports, qualitative, cross-sectional and cohort studies), with similarity in outcome themes. Results Following implementation of the search strategy, 38 relevant full texts were identified; comprising 12 case reports, 9 qualitative studies and 17 cohort studies, and data was also available from 24 relevant conference abstracts. No randomized studies were identified. Common themes were identified in the outcomes across the studies; glycemic variability, safety, quality of life, healthcare provider attitudes and social media. Conclusion There is extensive real-world data, but a lack of randomized control trial evidence supporting DIY AID system use, due to the user-driven, unregulated nature of these systems. Healthcare providers report a lack of understanding surrounding, and confidence in supporting, DIY AID despite impressive observational and user self-reported improvements in glycemic variability, without any reported safety compromises.

Introduction achieve the associated benefits of DIY AID, with the suggestion of hospitals or camps as potential safe locations to perform these novel studies [12]. More recently, in an attempt to rectify this uncertainty, the OPEN International Healthcare Professional Network and OPEN Legal advisory group have published a consensus statement, with practical guidance for HCP in the use of DIY or open-source AID [14]. This group, comprising specialist HCP and legal experts in the field of AID systems make reference to the challenges in supporting users of an unregulated system and the need to take into account local law and organizational governance in clinical practice.
We performed a scoping review of the currently available literature surrounding DIY AID systems, specifically to highlight the evidence available to surrounding their use. We aimed to identify studies reporting on the impact of DIY AID systems on type 1 diabetes management for both users and their care givers, or HCP, with the goal of collating outcome data; glycemic variability, safety and quality of life. In addition, we hoped to gain a greater understanding of the experiences of DIY AID system users and HCP providing care for people using these systems.

Methods
This review was carried out in accordance with the Enhancing Transparency in Reporting the Synthesis of Qualitative Research (ENTREQ) criteria [15] (Table 1.4, in S1 Appendix). A thematic synthesis methodology was used for analysis; with heterogeneity seen in the incorporated study designs (case reports, qualitative, cross-sectional and cohort studies), but consistency in the reported outcome measures. This methodology enables translation of concepts across studies, with organization of descriptive themes and transparency in analysis of the study results [16].

Data sources and searches
A review was conducted of the available literature in the use of DIY AID systems, published until 31 st December 2021. Relevant articles published in English were systematically sought using the databases Embase, Medline, Web of Science, Scopus, Proquest and Cochrane library. Terms were searched as keywords within the title or abstract of the manuscript, combining the description of the disease of interest (type 1 diabetes) and those used in the description of DIY AID systems (do-it-yourself, loop, automated insulin delivery or artificial pancreas system), the full search strategy is shown in Fig 1. Additional grey literature searches took place through the search engine 'Google', reviewing the first 10 pages of results in the search 'DIY' and 'type 1 diabetes' on 31 st December 2021. The chronological summary of DIY AID system outcome data available at openAPSoutcomes.org was additionally reviewed [4]. Conference abstracts were identified, in addition to those highlighted in the above searches, through review of the available listings for the past two years at the American Diabetes Association (ADA 2019, 2020), Diabetes UK (DUK 2020, 2021) and Advanced Technologies & Treatments for Diabetes (ATTD 2020, 2021) conferences, specifically to identify additional relevant data, not yet formally published.

Study selection
Two independent reviewers (AM and KC) screened the identified titles after implementing the search strategy (Fig 1), for those deemed to be relevant, review of abstract and full text (where available) took place. Article type and which AID system used were assessed. Case reports/case series, qualitative studies, prospective cohorts, retrospective cohorts, cross-sectional studies and conference abstracts were included for review (PRISMA Flowchart in Fig 2). Only studies in humans, published in English, reporting on the use of the DIY AID systems; OpenAPS, AndroidAPS or Loop were included in the results. Reference lists of both included studies and review articles were screened for any additional relevant studies requiring inclusion into this review.

Quality assessment
The identified cohort group DIY AID studies were sub-classified according to the data collection method; into qualitative, prospective and retrospective cohort or cross-sectional studies. Quality assessment of full text, excluding case report studies, was carried out using the Clinical Appraisal Skills Program checklist for qualitative and cohort studies (Tables 1.1 and 1.2 in S1 Appendix) and AXIS (the Appraisal Tool for Cross-Sectional studies) for the identified cross-sectional studies (Table 1.3 in S1 Appendix). These are validated checklists, according to study type, looking broadly at the domains of aims, methodology, results, analysis, overt findings and implications of these on future practice, with each domain assessed as 'yes', 'no' or 'can't tell' [17][18][19].

Analysis
For all relevant study and text types identified, reporting qualitative or quantitative results in the use of DIY AID systems, data was extracted on; first author, year of publication, geographical area, study methodology, participants, outcomes and measurement of these outcomes. Common themes were identified in the outcomes reported across the DIY AID cohort studies and the results reported within these outcome themes. The themes identified comprised; glycemic variability, safety, quality of life, healthcare provider attitudes and social media. Glycemic data are reported using a standardized TIR of 3.9-10.0 mmol/L, unless otherwise stated.

Results
Following implementation of the search strategy (Fig 1), 244 potentially relevant articles were reduced down to 70 full texts, after exclusion through screening of title and abstract (PRISMA,  Fig 2). Following detailed assessment, 62 studies reporting on the use of DIY AID systems were included in our results, these comprised; 12 case reports/case series, 17 cohort studies, 9 qualitative studies and 24 conference abstracts.

PLOS ONE
Quality assessment revealed high-quality study design across the included study types with quality consideration throughout study aims, methodology, recruitment processes, consideration of ethical issues, data analysis and reporting of findings. However, in 8 of the 9 reported qualitative studies it was unclear as to whether the relationship between the researcher and participant had been considered by the study team. Of the included cohort studies, 6 studies identified possible confounders in outcomes, but none of these considered confounders in data analysis. The key deficiency highlighted in cross-sectional study quality was a lack of consideration of response bias, identified in 3 of the 8 studies.
These 12 articles, report DIY AID use in a total of 20 individuals (70% female); 12 Loop, 5 AndroidAPS and 3 OpenAPS. The study outcomes report HbA1c (% or mmol/mol) in 10 studies, with HbA1c value pre-DIY system use additionally available in 5 of these studies, with 80% reporting an improvement in HbA1c with DIY AID use. TIR is a reported outcome in 10 of the studies, with pre-DIY AID TIR values available in 2, and an improvement in TIR with DIY AID use demonstrated in both of these. Diabetes related quality of life score (dQOL) and fear of hypoglycaemia (HFS-II) were reported outcomes in one case report, with improvements seen with Loop use [28].
Case reports additionally discussed the beneficial impact of DIY AID use in people with type 1 diabetes in situations where maintaining stable blood glucose levels would be challenging; running a half marathon [23] and fasting during Ramadan [26].
HCP opinions on DIY AID systems were reported in 4 studies, and 15 studies reported user opinions and/or their outcomes with DIY AID use. The USA contributed to the production of the greatest number of cohort studies (n = 5, 29%), in addition to Germany (n = 3), United Kingdom (n = 3), Czech Republic (n = 2), Switzerland (n = 1), Australia (n = 1), China (n = 1) and Poland (n = 1).
The majority of these studies (n = 6) reported on the outcomes with a combination of the three DIY AID system types. Participants included; users (5 studies), care givers (2 studies), HCP (2 studies), mentors in the DIY AID community (1 study, 9 participants), people with type 1 diabetes not currently using a DIY AID system (1 study, 16 participants), as well as people that had decided to stop using Loop (1 study, 45 participants).
Glycemic variability. Self-reported retrospective user data looking at glycemic variability with OpenAPS use was first published by Lewis in 2016, with 18 participants reporting improved HbA1c (mean 6.2 vs 7.1% prior to OpenAPS use) and TIR (80-180mg/dL); (81 vs 57% prior to OpenAPS) [32]. Further data on OpenAPS outcomes was published by Melmer in 2019, through retrospective analysis of 80 users' CGM data, that had been uploaded to the OpenAPS Data Commons Repository. Of these, 34 users had additional data available from Sensor-augmented insulin pump (SAP) use prior to OpenAPS; with a mean reduction in HbA1c of 0.4% (p<0.0001) and increase in TIR of 9.3% (p<0.0001), relative to SAP use [35].
Sole use of AndroidAPS was studied in 4 of the observational studies, both retrospective (n = 2) and prospective (n = 2), including a total of 85 participants. AndroidAPS implementation ranged from a minimum of three nights, up to six months duration [34,41,43,44]. The three studies with a minimum of three months AndroidAPS use, all reported improvements in HbA1c and TIR from baseline. The largest of these by Petruzelkova et al., followed 36 children; 18 pre-school (age 3-7 years) and 18 school age (age 8-14 years), for six months following switching from SAP to AndroidAPS. Glycemic outcomes improved in the pre-school children    [43]. With Loop, one prospective observational study followed 558 participants for six months, after initiation of Loop. TIR (70-180mg/dL) and HbA1c were compared from baseline therapy, to after six months of Loop use with improvement seen in both of these parameters with Loop in comparison to baseline therapy; mean TIR 73 vs 67% (p<0.001) and HbA1c 6.5% vs 6.8% (p<0.001) [42].
Safety. The frequency of episodes of hypoglycemia (n = 2), severe hypoglycemia (n = 2) and TBR (n = 7) were reported in a total of 9 studies [33][34][35][40][41][42][43][44][45], with improvements in these parameters reported with the use of all three DIY AID system types. TBR was reduced by 0.1-1.11% across the observational studies [41,42,44]. Two observational studies reported no improvement in TBR [34 and 43] using Android APS, time spent 3.0-3.8mmol/L remained static at 3% in pre-school age children, and increased from 2.6 to 3.8% in school age children using AndroidAPS for six months, relative to baseline SAP [43]. An increased TBR, with three nights AndroidAPS use in 22 children was reported relative to SAP; 5% vs 3% [34]. In user self-reported studies (n = 2), 17.9% [40] and 74% [33] of users reported a reduction in the frequency of hypoglycemia with the use of DIY AID. Gawrecki et al., reported primary outcomes of both safety and glycemic control. No cases of Severe Hypoglycemia or Diabetic Ketoacidosis (DKA) occurred in twelve weeks of Androi-dAPS use. A reduction in percentage time spent <54mg/dL (0.1%), and percentage time <70mg/dL (0.75%), was demonstrated in 12 AndroidAPS users, relative to baseline CSII [44]. Safety was assessed in 68 AID users, comparing outcomes in 38 users of Commercial and 30 DIY AID. Over six months no episodes of DKA occurred and TBR was 3.2% with DIY and 2.6% Commercial AID. This study highlighted a non-significant increase in severe hypoglycemia in users of Commercial relative to DIY AID (p = 0.104) [45].
Quality of life. Quality of life was assessed by Wu et al. in 15 participants with 3 months of AndroidAPS use, through the use of the EuroQol Five-Dimension 5-Level Health Questionnaire (EQ-5D-5L), both in the form of utility index (UI) and visual analogue scale (VAS). Improvement in mean score with AndroidAPS use, was seen in VAS relative to baseline; 82 vs 77 (of a maximum 100), p = 0.13, but no change demonstrated in UI, mean 0.88 vs 0.88 (of a maximum 1). With UI calculated on a 0 to 1 scale (no to severe impairment/unable) scored through questions in the five dimensions of mobility, self-care, usual activities, pain/discomfort and anxiety/depression. A small improvement was seen in diabetes-related distress; an increase from 6 to 9% (p = 0.143) in those scoring little or no diabetes distress, and a reduction in fear of hypoglycemia relative to baseline therapy through use of the Hypoglycemia Fear Survey II-Worry Scale (HFS-II); mean score 22.13 vs 26.27 (lower score better, of a maximum 72),

Self-Reported
Lewis ( PLOS ONE p = 0.01 [41]. Self-reported improved sleep quality was highlighted by 94% of OpenAPS users [32], and 79% of DIY AID users [33]. When questioned regarding the main benefits of DIY AID, 22.5% of 86 participants, reported better sleep quality/nightly safety [40]. Assessment of the motivations to commence DIY AID, revealed 72% of adults and 80% of caregivers cited improved sleep quality as a motivating factor for this choice of insulin delivery system [5]. An abstract for ADA by Hood et al., assessed quality of life outcome measures both before and after 3 months of using Loop, in 254 new Loop users. Improvements were seen in mean scores of diabetes-related distress, as measured by the diabetes distress scale (DDS); 2.06 to 1.66 (scored from 1; not a problem, to 6; a very significant problem). A reduction in fear of hypoglycemia (HFS-II); 19.74 to 17.18, and improvement in sleep were also demonstrated, (Pittsburgh Sleep Quality Inventory, scored 0 to 21); 6.82 to 5.39 [66]. In a cross-sectional study abstract for ATTD, Zabinsky et al. explored self-reported outcomes with DIY AID in 180 users; 74.7% reported improved sleep quality/quantity, 69.4% reduced time spent managing diabetes and 76.9% reduced diabetes-related stress [67].
Healthcare provider (HCP) attitudes. HCP opinions on DIY AID use were collected in 4 studies, 3 from the USA and one from the UK, with a total of 753 HCP surveyed [37][38][39]46]. In the UK study of 317 HCPs, 91% of participants advised they would not initiate discussions about DIY AID and 2% reported that they perceived the systems as dangerous [38]. One study from the USA, surveyed 152 HCP approached via the American Association of Diabetes Educators, 27% reported that they perceived these systems as safe [39]. A lack of understanding in how the systems work was reported by 74.4% of participants, 11.6-34.9% felt comfortable answering questions about DIY AID systems [37] and 97% reported a willingness to learn more about them [38]. Fear of HCP disapproval of DIY AID was reported as a prominent reason for users who had decided to stop using Loop [56]. In a survey of 104 school nurses, 23% reported a child using DIY AID attended their school, 46% stated they had no prior knowledge of DIY AID and 96% felt the child should be able to share their data with a parent or guardian during the school day [46]. HCP supporting AndroidAPS use as part of the Community Derived Automated Insulin Delivery study (CREATE) in New Zealand [81,82], found that user challenges with this system most frequently related to device issues (the insulin pump and cannula in 24% of analyzed conversations), as opposed to DIY AID specific challenges [54].
Cohen et al. reported a qualitative interview study of the perceived benefits and barriers of DIY AID use in 20 HCP working in pediatric and adult diabetes services in the UK, as an abstract for ATTD. Of the 20 participants, 19 reported liability concerns and lack of formal guidelines to be barriers to supporting the current use of DIY AID in widespread clinical practice [76].
Social media. Attitudes from users and the DIY AID community were collected through 6 studies using social media (Facebook and Twitter), with 2 reporting the content of Tweets [48,51], and 4 using these platforms to distribute surveys to DIY AID users [33,36,39,47]. A total of 49,925 tweets were analyzed from 8214 participants [48,51]. User opinions of DIY AID across these studies were positive, with 82-85% positive interactions on Twitter [48]. Through an interview study, mentors in the DIY AID community, largely through the use of social media platforms, reported altruism as the main reason behind their role, and the frequency of questions and managing workload, to be the biggest challenges they face [50].
Of the conference abstracts reviewed, 2 detailed the recruitment of participants through the use of social media. A qualitative interview study of 11 girls and women, discussing glycemic variability and need for algorithm adjustment relating to hormonal changes, recruited study participants through topic related discussion on social media [75]. Girelli et al., distributed a survey via the Looped and OpenAPS Facebook groups, gaining responses from 120 respondents interested in DIY AID use and 19 current users [70].

Discussion
The findings of this scoping review highlight that there is a large, and rapidly expanding body of published outcome data relating to DIY AID system use. We have reviewed an extensive scope of outcome studies, varying in aims and methodology. With the nature of research into these rapidly evolving systems, there is no doubt that at the date of publication of this review, new insights into the field will already be available. This may be particularly apparent with the lack of DIY system subtype specific terminology in our search terms (Android or OpenAPS), and especially in the use of newer DIY system types (including FreeAPS and AI APS). However, through this review we have highlighted evidence of impressive and consistent glycemic outcomes with all forms of DIY AID use; improvements in TIR, HbA1c and TBR, and have seen no great discrepancies between the outcomes reported in observational studies relative to self-reported data [5,32,35,36,[41][42][43][44][45]47].
Social media platforms have provided both a source to gather large quantities of user data, as well as a means for recruitment of study participants. In addition, we have found online resources and social media, especially Facebook and Twitter, to be a large part of the support structure for people using DIY AID [50,53], with the Looped Facebook group now having over 28,000 members [83]. Supporting safety and efficacy data are additionally published by users themselves through internet resources, notably in the form of blogs [84]. This online community however, is not a conventional method to collect and publish data relating to pharmaceutical and technological advances, and will bring the validity of the reported results into question, with the prospect of selection bias in addition to the seemingly very favorable user outcomes being self-reported.
Current users of DIY AID systems are in themselves self-selected, the systems are user-built and individualized. The development of the technology in DIY AID has been driven by the user from its outset, rather than a pharmaceutical company, making it challenging to collect impartial and externally valid data on its use, as well as to fund large clinical trials [3]. This individualized user choice in method of insulin delivery for management of type 1 diabetes, brings into question both the appropriateness and the utility of randomized control trials (RCT), in the setting of these systems.
It is challenging to compare results between the studies reviewed, as well as between the DIY AID system types, due to the variable design and duration of the studies described, in addition to the inclusion criteria for participants. We have reviewed observational data relating to the use of AndroidAPS (85 participants) [34,41,43,44], with three of these studies implementing the system for a minimum of three months (63 participants), all demonstrating improvement in mean HbA1c and TIR [41,43,44]. An observational study of Loop (558 participants) from the USA highlighted similar improvements in HbA1c (0.3%), TIR (7%) and a reduction in TBR (0.1%). Notably these observational improvements in glycemic outcomes, were all within in the first six months of Loop use, in individuals with already close to optimal glycemic control; mean HbA1c at baseline 6.8% [42]. These glycemic outcomes are not representative of the 'average' person with type 1 diabetes, with just 21% of American adults with type 1 diabetes reaching an HbA1c of less than 7% (53mmol/mol) [85]. This reiterates the concept that users of DIY AID do not represent a 'typical' person with diabetes, and that the outcomes demonstrated with these systems in observational studies are largely a reflection of the individual that is choosing to use this technology, in conjunction with the benefits of the system itself. There is questionable generalizability of the results across the studies we have reviewed, to the implementation of these systems in the wider population of people with diabetes. Despite the overwhelmingly positive outcomes in both the observational studies and self-reported data, we cannot infer from this that these systems are the optimal glucose management system for all people with type 1 diabetes.
In order to set up these seemingly very beneficial systems, a level of understanding of both the technology and type 1 diabetes management is needed. DIY AID does not remove the requirement for user input in the management of type 1 diabetes; understanding of carbohydrate counting, insulin:carbohydrate ratios and insulin sensitivity factors, the technology alone is not enough [86]. In addition to diabetes education, a level of both literacy and numeracy skills are needed to follow the instructions for system set up and to overcome any barriers that may be faced in this process [87][88][89]. These resources guide users in customisation of settings, with variable DIY AID algorithm types available to meet user needs [1]. The studies reviewed highlight the importance of internet resources, social media platforms as well as mentors in the DIY AID community in guiding users through the set up and any on-going challenges with these systems. This community of users and advisors have much greater experience in the use of these systems than the majority of HCP [8,83].
Currently, the appropriate role of HCP in DIY AID is not clear and the studies of HCP knowledge and attitudes in the use of these systems reflect this uncertainty. Due to the novelty and rapidly evolving nature of these systems, the majority of HCP caring for people with diabetes have not received any formal system-specific education whilst training for their roles. With the rising popularity of DIY AID worldwide, the majority of HCP working in the specialty of Endocrinology and Diabetes, are now likely to be the responsible clinician for one or more person using some form of DIY AID [8]. With a lack of training in their use, as well as unresolved ethical and potentially medico-legal concerns, it is unsurprising that the majority of HCP are not voluntarily broaching the subject of DIY AID systems in consultations with people with type 1 diabetes [38]. A lack of system-specific training, and open communication with potential future as well as current DIY AID users, results in compromised and inconsistent care for people with diabetes. The OPEN consensus statement was not available at the time the studies we have reviewed took place [14]. Further HCP opinion studies going forward may reflect greater confidence in discussing this potential management option with people with diabetes and/or their caregivers, in line with this guidance. However, there are increasing capabilities and available options in Commercial forms of AID, if these ongoing advancements meet user needs there may be a reduction in future DIY AID use, with HCP potentially more likely to actively support users choosing a regulatory approved glucose management system. Active support of an 'off-label' medical device, such as DIY AID, requires strong clinical evidence which is, defined as a minimum of one RCT in Canada [90]. Despite the broad scope of evidence to support DIY AID use highlighted by this review; beneficial glycemic outcomes, quality of life measures and supportive safety data, no RCT has been completed. An RCT protocol has been completed in New Zealand; comparing six months of Android APS to Sensor Augmented Pump therapy, with results currently awaited [81]. This study was funded by the Health Research Council of New Zealand and includes both children and adults [82]. The OPEN consensus statement makes reference to the difficulties we have discussed, in performing an RCT in the use of DIY AID, suggesting the extensive real-world data available, that we have highlighted in this scoping review, should be considered in regulatory approval processes [14].

Conclusion
There is a vast and rapidly expanding body of observational and self-reported user data available in the use of DIY AID systems. There are however substantial potential weaknesses in these studies, with inclusion of biased samples presenting challenges in the generalizability of this real-world evidence. No RCT data is currently available for any of the DIY AID system types; the standardized method to achieve conclusive, unbiased, level 1 evidence. This may however not be the optimal data collection methodology in the assessment of outcomes with this self-selecting, individualized and user-built technology choice. In contrast to the evidence supporting commercially available AID systems, an objective unbiased data deficiency for DIY AID and lack of regulatory approval is resulting in uncertainty worldwide among HCP. Education and best practice recommendations for HCP are lacking, in the utilization of DIY AID systems. These interventions are imperative to enable appropriate and optimal HCP support for people with type 1 diabetes choosing to use these glucose management systems. Mentors within the DIY AID community have been highlighted as rich knowledge sources, who could play a key and essential role, in developing focused education and training programs, in this exciting and rapidly expanding field.